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Abstract 


These  guidelines  are  intended  as  a resource  for  members  of  the  economic  evaluation  community 
who  wish  to  perform  economic  evaluation  studies  of  the  Advanced  Technology  Program  (ATP). 
To  help  members  of  this  community  frame  the  performance  evaluation  problem  in  the  context  of 
the  ATP,  background  information  is  provided  on  the  program,  on  its  evaluation  efforts  and 
accomplishments  to  date,  as  well  as  its  evaluation  objectives  and  principal  areas  of  interest.  An 
overview  is  provided  of  ATP’s  procedures  for  considering  proposals  on  program  evaluation  and 
the  criteria  against  which  proposed  studies  are  scored.  Suggestions  are  given  for  the  content  and 
format  of  proposals  and  reports  on  evaluation.  The  guidelines  in  no  way  constitute  a solicitation 
for  proposals  on  evaluation  or  an  obligation  to  fund  any  proposals  that  might  be  submitted;  they 
are  provided  to  improve  the  quality  and  relevancy  of  evaluation  studies,  and  the  efficiency  and 
effectiveness  of  the  ATP/evaluator  interface. 

Keywords 

advanced  technology  program;  benefit-cost  analysis;  economic  analysis;  evaluation;  impact 
assessment;  performance  metrics;  research  and  development;  technology  assessment 
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Executive  Summary 


The  Advanced  Technology  Program  (ATP)  is  one  component  of  the  nation’s  strategy  to  promote 
the  economic  well-being  of  the  country.  The  ATP  encourages  industry  to  accelerate  the  creation 
and  commercialization  of  enabling  technologies  that  are  expected  to  yield  large  economic  benefits 
to  the  nation,  extending  significantly  beyond  the  direct  benefits  to  award  recipients.  The  ATP 
provides  this  catalyst  through  research  awards  to  companies  for  overcoming  challenging  technical 
barriers  that  currently  stand  in  the  way  of  realizing  the  potential  benefits;  in  fact,  ATP  funding 
goes  only  for  research  that  presses  the  scientific  or  technical  state-of-the-art  in  the  given  industry 
sector.  Businesses  conceive,  propose,  and  carry  out  the  research  projects  for  advanced 
technology  development  that  are  cost-shared  by  the  ATP.  The  ATP  awards  are  made  to 
individual  companies  and  to  research  joint  ventures  --  comprised  of  two  or  more  companies,  often 
in  combination  with  universities  and  non-profit  research  laboratories  ~ in  intense  competition. 

The  ATP  held  its  first  competition  in  1990,  and  to  date  has  funded  280  projects  totaling  about  $2 
billion  in  advanced  research,  with  more  than  half  the  funds  coming  from  industry  in  cost  share. 
Some  of  the  early  projects  are  now  moving  into  the  commercialization  phase,  but  most  of  the 
projects  were  funded  in  the  last  two  years  and  are  still  in  the  research  phase.  Though  it  is  still 
early  in  the  program,  there  is  considerable  interest  in  the  likely  outcomes  — both  to  meet  internal 
management  requirements  for  information,  and  to  satisfy  external  policy  related  demands. 

The  ATP  has  emphasized  program  evaluation  from  the  outset.  To  this  end,  the  ATP  utilizes  the 
assistance  of  highly  qualified  academic  and  consulting  economists  and  other  experts  in  evaluation, 
in  addition  to  its  in-house  staff  --  in  planning,  modeling,  and  developing  databases,  in  conducting 
surveys,  case  studies,  statistical  and  econometric  analyses,  and  in  carrying  out  other  studies  that 
enhance  program  performance  and  contribute  to  the  quality  and  reliability  of  performance  metrics. 

This  summary  report  is  provided  to  help  the  economics\evaluation  communities  better  understand 
the  ATP  and  its  on-going  activities  in  evaluation.  The  goal  is  to  provide  an  efficient  and  effective 
means  of  communicating  to  those  communities  what  the  ATP  is  doing  and  what  it  wishes  to  do  in 
the  field  of  evaluation.  All  the  economic  evaluation  work  that  the  ATP  supports  is  needs-driven 
and  geared  either  towards  increasing  the  expected  net  benefits  of  the  program  or  measuring  the 
net  benefits,  or  both.  Economic  research  of  a general  nature,  not  directly  relevant  to  the  ATP,  is 
not  supported.  The  guidelines  provided  herein  are  for  researchers  proposing  and  conducting  ATP 
evaluation  work.  The  document  is  merely  explanatory  and  does  not  constitute  a solicitation  for 
proposals. 

To  help  researchers  frame  the  evaluation  problem  in  the  context  of  the  ATP,  background 
information  is  provided  on  the  program  and  its  evaluation  goals,  activities,  accomplishments,  and 
principal  areas  of  interest.  A model  of  the  program  is  provided  as  a framework  for  thinking  about 
the  evaluation  needs  of  the  ATP.  Researchers  are  reminded  of  the  importance  of  tying  proposed 
evaluation  studies  directly  to  ATP’s  mission  and  project  goals;  of  bridging  the  gap  between 
interesting  theoretical  work  and  the  need  for  practical  relevancy  to  ATP’s  evaluation  problems. 
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The  ATP  expects  that  the  evaluation  studies  it  funds  will  not  only  further  the  understanding  of  the 
program  directly,  but  will  push  the  state-of-the-art  in  evaluation,  resulting  in  tools,  methods,  and 
data  of  broad  interest  and  leading  to  publishable  papers  in  refereed  journals.  Examples  of 
evaluation  topics  of  current  interest  to  the  ATP  include  predicting  and  measuring  spillover  effects; 
assessing  the  benefits  and  costs  of  collaboration;  understanding  organizational  effects  of  projects 
on  participating  firms  that  may  extend  beyond  the  project  walls;  R&D  financing  issues  and  the 
impact  of  the  ATP  on  private  investment  in  R&D;  the  development  of  new  models  and  tools  for 
measuring  inputs  and  impacts;  as  well  as  other  topics.  Recommendations  are  given  for  the  format 
and  contents  of  proposals  and  reports.  From  time  to  time,  the  ATP  plans  to  issue  a supplement  to 
the  Guidelines  updating  evaluation  studies  underway  and  completed.  Users  of  the  Guidelines  are 
invited  to  share  their  advice  on  ways  to  improve  the  document’s  usefulness  as  a reference  for 
evaluation  researchers  interfacing  with  the  ATP. 
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Guidelines  for  Economic  Evaluation  of  The  Advanced  Technology  Program 


1.  Evaluation  Studies  in  the  Advanced  Technology  Program  (ATP)  Context 

The  ATP’s  mission  is  to  partner  with  U.S.  businesses  in  high-risk  scientific  and  technical  research 
to  develop  enabling  technologies  with  strong  potential  for  producing  broad  economic  benefits  for 
the  U.S.  It’s  activities  in  the  fields  of  economics  and  evaluation  are  purely  in  support  of  this 
mission.  The  ATP  does  not  fund  general  or  basic  economic  research  outside  of  this  scope. 

In  the  context  of  its  mission,  the  ATP  has  evaluation  efforts  underway  in  support  of  increasing 
and  measuring  the  short-  and  long-run  impacts  of  the  technology  development  projects  it  funds, 
and,  ultimately,  of  the  entire  program.  As  part  of  that  effort,  the  ATP  utilizes  the  assistance  of 
academic  and  consulting  economists  and  other  experts  in  evaluation,  in  addition  to  its  small  in- 
house  staff  of  economists.  The  assistance  of  outside  experts  is  used  in  its  planning  activities, 
development  of  evaluation  methods,  models,  and  databases,  conducting  surveys,  case  studies, 
statistical  and  econometric  analyses,  and  in  carrying  out  other  studies  that  enhance  its 
performance  and  contribute  to  the  quality  and  reliability  of  its  performance  metrics. 

1.1  Relevancy  to  the  ATP 

Although  it  is  the  ATP’s  hope  that  the  studies  it  funds  will  contribute  significantly  to  the  broad 
fields  of  R&D  economics  and  performance  evaluation,  and  be  of  keen  interest  to  those  working  in 
these  fields,  the  real  purpose  of  ATP-fiinded  studies  is  to  advance  the  understanding  of  the 
workings  and  outcomes  of  the  ATP.  It  is  expected  that  researchers  will  be  well  informed  about 
the  ATP,  that  evaluation  topics  will  have  direct  relevancy  to  the  ATP,  and  that  researchers  will 
clearly  draw  potential  implications  of  their  research  for  the  ATP  in  some  detail.  It  is  expected  that 
successful  studies  funded  by  the  ATP  will  both  satisfy  the  researchers’  interest  in  pressing  the 
state-of-the-art  of  evaluation  — leading  to  publishable  papers  in  peer-reviewed  journals  — and 
satisfy  the  ATP’s  need  for  better  evaluation  tools  and  more  reliable  performance  metrics.  It  is 
expected  that  ATP’s  management,  policy  makers,  the  broader  R&D  community,  the  economics 
community,  and,  in  fact,  all  those  interested  in  performance  evaluation  will  benefit  from  the 
resulting  studies. 

1.2  In  Tune  with  ATP’s  Mission 

To  evaluate  a program  or  project,  it  is  essential  to  understand  the  program  mission  or  project 
goals,  and  to  tailor  the  evaluation  accordingly.  Thus,  proposers  of  evaluation  studies  are  strongly 
encouraged  to  take  the  time  to  review  and  understand  the  ATP’s  mission.  Researchers 
performing  case  studies  of  given  projects  and  sets  of  projects  in  focused  programs  are  strongly 
encouraged,  in  addition,  to  review  their  particular  goals  and  objectives. 
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1.3  ATP’s  Mission 


The  ATP  was  created  by  the  Technology  Competitiveness  Act  of  1988  and  received  its  first 
budget  in  1990.  According  to  the  Statute  establishing  the  ATP,  the  program  was  designed 
specifically  for  the  purpose  of  “assisting  United  States  businesses  in  creating  and  applying  the 
generic  technology  and  research  results  necessary  to  (1)  commercialize  significant  new  scientific 
discoveries  and  technologies  rapidly,  and  (2)  refine  manufacturing  technologies.”  It  was  further 
charged  to  “aid  industry-led  United  States  joint  research  and  development  ventures...  including 
those  involving  collaborative  technology  demonstration  projects  which  develop  and  test  prototype 
equipment  and  processes....”  The  Statute  emphasized  that  the  funding  should  go  “to  support 
projects  which  are  high  risk  and  which  have  the  potential  for  eventual  substantial  widespread 
commercial  application.”  It  further  directed  that  “the  Program  focuses  on  improving  the 
competitive  position  of  the  United  States  and  its  businesses,  gives  preference  to  discoveries  and  to 
technologies  that  have  great  economic  potential,  and  avoids  providing  undue  advantage  to 
specific  companies.”1 

The  ATP’s  Rule2,  developed  to  implement  the  program,  and  other  program  descriptive  materials, 
further  define  the  program.  The  Rule  emphasizes  that  the  ATP-funded  technologies  must  be 
“enabling”  in  that  they  must  offer  a “wide  breadth  of  potential  application  and  form  an  important 
technical  basis  for  future  commercial  applications;”  and  they  must  be  “high  value,  because  when 
applied,  they  offer  significant  benefits  to  the  U.S.  economy.”  Recent  ATP  program  descriptions 
define  “enabling  technologies”  in  terms  of  their  “potential  to  generate  economic  and  technical 
opportunities  that  stimulate  broad-based  benefits  for  the  nation,  beyond  those  accruing  directly  to 
ATP  awardees.”  Enabling  technologies  are  further  categorized  as  “pathbreaking,  infrastructural, 
and/or  multi-use.”3  Pathbreaking  technology  is  described  as  inducing  revolutionary  change  in 
existing  fields  or  opening  up  new  fields  of  activity.  Infrastructural  technology  is  described  as 
supporting  the  R&D,  production,  and  business  of  entire  industries.  Multi-use  technology  is 
described  as  having  many  distinct  applications.4  “High-risk  technologies”  are  defined  by  the  ATP 
as  “technical  challenges  which  display  significant  recognized  uncertainty  of  success,  where 
success  will  dramatically  change  the  future  direction  of  technology  and  its  market  impact.”5  Note 
that  while  the  ATP  is  constrained  to  funding  research  that  is  deemed  to  be  high-risk  in  the  sense 
that  the  scientific  and  technical  work  is  highly  challenging  and  fought  with  difficulty,  it  prefers,  all 
else  being  equal,  low  business-risk  circumstances. 


Omnibus  Trade  and  Competitiveness  Act  of  1988  (P.  L.  100-418,  15  U.S.C.  278n),  as 
amended  by  the  American  Technology  Preeminence  Act  of  1991  (P.L.  102-245). 

215  Code  of  Federal  Regulations,  Part  295,  Subpart  A,  Sec.  295. 

3 1996  ATP  Bidders’  Conference  Materials. 

AIbid. 

5Ibid. 
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In  summary,  the  ATP  enters  into  partnerships  with  industry  to  create  and  commercialize  rapidly 
high-risk,  enabling  technologies  that  have  the  potential  to  improve  substantially  the  productivity 
and  the  competitiveness  of  U.S.  businesses  and  to  yield  large  economic  benefits  to  the  nation. 
Reflecting  the  goal  of  broad-based  benefits  from  the  program  extending  beyond  the  direct  award 
recipients,  the  ATP  emphasizes  the  importance  of  a strong  potential  for  spillover  benefits  from  the 
technologies  it  funds. 

1.4  Overview  of  ATP  Partnerships  with  Industry 

Industry  leads  in  the  ATP-style  of  industry/govemment  partnerships  for  advanced  technology 
development.  Industry  conceives,  proposes,  and  carries  out  the  research  projects,  and  undertakes 
subsequent  commercialization  activities.  The  ATP  evaluates  technology  development  proposals 
submitted  to  it  for  their  technical  and  business\economic  merits  in  rigorous  competitions.6  It 
makes  awards  to  the  top-scoring  projects,  and  subsequently  monitors  technical  and  business 
progress  throughout  the  project  duration.  If  desired  by  the  companies  and  to  the  extent  feasible 
and  appropriate,  the  ATP  project  monitor  will  arrange  NIST  laboratory  support  or  use  of  special 
government  laboratory  facilities  to  overcome  tough  technical  obstacles  standing  in  the  way  of 
success;  hence,  ATP  awards  may  entail  scientific/technical  assistance  as  well  as  dollars. 

Funding  assistance  is  provided  in  the  form  of  government  sharing  of  “allowable  costs”  — research 
costs  only,  exclusive  of  product-development  costs  — through  the  formation  of 
govemment/industry  cooperative  agreements.  These  arrangements  entail  a more  active  role  for 
government  than  a grant.  They  also  differ  from  a contract  in  that  the  government  is  not  a 
customer  for  project  output.  Contracts  and  grants  are  typically  NOT  used  by  the  ATP  to  fund  its 
technology  development  projects.  The  ATP  funds  the  research  on  a cost-reimbursable  basis, 
quarter-to-quarter.  Either  the  company  awardees  or  the  ATP  may  terminate  a project  at  any  time. 

Companies  can  propose  projects  in  ATP’s  technology  development  competitions  either  as  single- 
company applicants  or  as  research  joint  ventures.  There  is  a cap  on  single-company  awards  of  $2 
million  and  3 years.  Joint  ventures  are  not  subject  to  a funding  cap  and  the  projects  may  last  up  to 
5 years.  The  single-company  award  recipients  must  cover  all  of  their  indirect  costs  and  may  cover 
some  of  their  direct  costs;  the  joint-venture  participants  must  provide  more  than  half  the  total 
project  costs. 

A joint  venture  at  a minimum  must  have  two  for-profit  companies  performing  research  and 
contributing  to  the  cost-share.  Many  joint  ventures  involve  more  than  two  members.  Proposing 


Application  guidelines  and  selection  criteria  are  presented  in  ATP  Application  Kits  which 
are  updated  and  reissued  periodically.  As  of  October  1996,  the  program  was  using  a kit  dated 
November  1994,  in  combination  with  a supplement  dated  May  1996. 
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companies  can,  at  their  discretion,  include  other  companies,  universities,  non-profit  research 
laboratories,  associations,  and  others  in  their  projects,  either  as  formal  partners  in  a joint  venture 
or  as  informal  partners  or  subcontractors  in  either  a joint  venture  or  a single-company  project. 

The  ATP  is  NOT  a funder  of  streams  of  basic  research  in  the  traditional  mode  of  the  National 
Science  Foundation.  It  is  NOT  a procurer  of  mission-related  applied  research  in  the  mode  of  the 
Defense  Department  and  the  Department  of  Energy.  It  funds  research;  not  product  development. 
It  operates  in  the  middle  ground  between  basic  research  and  product  development. 

The  ATP’s  focus  is  on  civilian  technologies  — not  defense  technologies.  The  ATP  is  normally 
NOT  a customer  for  the  results  of  the  technology  development  projects  it  funds.  The  ultimate 
commercial  outcomes  are  the  results  of  the  marketplace  interplay  of  demand  and  supply  forces. 
The  ATP  looks  closely  in  project  proposals  at  proposers’  arguments  that  they  intend  to  pursue 
future  commercialization  of  the  technologies,  and  assesses  the  strength  of  their  proposed 
pathways  to  market  and  their  commitment  to  follow  through  with  commercialization  plans.  It 
relies  on  the  presence  of  expected  private  returns  to  induce  companies  to  be  willing  to  cost-share 
the  research  with  the  ATP  and  subsequently  to  carry  out  commercial  development  of  the  new 
technology  with  private  capital.  But  it  selects  only  those  projects  for  awards  for  which  it  thinks 
the  potential  social  rate  of  return  (the  return  to  the  nation)  far  exceeds  the  potential  private  rate  of 
return  on  investment,  and  for  which  it  thinks  the  private  sector  will  either  not  do  the  project  at  all, 
or  not  within  the  critical  time,  or  in  the  scale/scope,  necessary  to  realize  the  potential  societal 
benefits. 

In  absence  of  intervention,  these  research  “spillovers”  (or  externalities),  that  accrue  to  others  than 
the  party  that  undertakes  the  research,  are  generally  ignored  in  private  investment  decisions, 
resulting  in  a type  of  “market  failure”  characterized  by  an  underinvestment  in  R&D  from  a 
national  perspective.  Capital  market  imperfections  may  be  an  additional  factor  that  results  in  the 
funding  of  a less  than  socially  desirable  level  of  research.  The  ATP  is  one  of  the  national  policy 
strategies  for  overcoming  R&D  market  failure.  A successful  ATP  will  in  the  long-run  result  in  net 
societal  benefits  greater  that  would  have  resulted  without  it. 

1.5  A Three-Dimensional  Model  of  the  ATP7 

Key  characteristics  of  the  ATP  have  been  captured  by  J.-C.  Spender  in  a three-dimensional  model 
of  the  program  (Spender,  The  Three  Dimensional  Model  of  the  Advanced  Technology  Program , 
NIST  report  in  draft,  October,  1996).  The  model,  depicted  in  figure  1,  measures  on  the  O-TK 
axis  the  success  of  ATP-funded  technology  development  projects  to  increase  the  level  of  scientific 
and  technical  knowledge  (by  overcoming  high-risk  technical  challenges).  It  measures  on  the  O- 
PR  axis  the  net  private  returns  on  investment  captured  by  ATP  project  participants.  And,  thirdly, 
it  measures  on  the  O-PG  axis  the  net  spillover  returns  to  others  in  the  economy  who  benefit  from 


7The  3-D  model  of  the  ATP  was  developed  by  J.-C.  Spender,  and  is  described  in  a 
forthcoming  report  by  Prof.  Spender. 
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the  project  either  through  consumer  surplus  benefits  derived  by  purchasing  improved  or  cheaper 
goods  and  services  at  prices  that  do  not  fully  capture  their  increased  value,  or  by  utilizing  the 
knowledge  derived  from  the  ATP  project  to  generate  additional  economic  benefits  without  fully 
compensating  the  knowledge  generators.  The  vectors  mapped  by  OTPQ  in  the  figure  below 
describe  a fully  successful  ATP  project  that  delivers  a combination  of  high-risk  technical 
accomplishments,  positive  net  private  benefits,  and  positive  net  spillover  benefits.  The  ATP  seeks 
to  maximize  net  social  benefits  (defined  here  as  benefits  to  the  nation)  from  the  program,  subject 
to  the  constraints  that  the  state  of  the  art  be  pressed,  the  effort  be  led  and  carried  out  by  U.S. 
industry,  and  the  benefits  be  “broad-based.”  In  the  parlance  of  ATP,  the  program  seeks  with  each 
project  it  funds  a “journey  to  Q.”  By  encouraging  companies  to  increase  their  development  of 
enabling  technologies  that  will  strengthen  the  nation  economically,  the  ATP  represents  a new 
streamlined,  efficient  model  for  government,  whereby  government  facilitates  rather  than  directs; 
taps  industry’s  resources  and  know-how  rather  than  bureaucracy  to  get  the  job  done. 

Figure  1. 

ATP's  Evaluation  Program 
Designed  for  ATP's  Three  Dimensions 


Note:  Based  on  charts  presented  by  J.-C.  Spender  in  a forthcoming  report. 
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2.  Overview  of  ATP’s  Evaluation  Program 

2.1  External  Demands  for  Evaluation  of  ATP 

Externally,  the  ATP  faces  intense  demands  for  performance  measures.  Many  requests  are  driven 
by  policy  issues.  Specific  requests  for  evaluation  results  come  frequently  from  individual 
members  of  Congress  and  their  staff,  from  Congressional  subcommittees,  the  General  Accounting 
Office,  the  Executive  Office  of  the  President,  the  Office  of  Management  and  Budget,  the  Office  of 
Inspector  General,  the  Press,  think  tanks,  industry,  and  others. 

The  ATP  also  receives  many  external  inquiries  about  its  evaluation  program  that  are  focused  more 
on  evaluation  tools  and  methodologies  than  on  specific  evaluation  results.  For  example,  inquiries 
come  from  counterpart  programs  in  other  countries,  from  other  Federal  agencies,  State  and 
regional  government  agencies,  universities,  businesses,  and  consultants  who,  like  the  ATP,  are 
concerned  about  the  ins  and  outs  of  performance  metrics  and  measurement  methodologies.  It  is 
the  ATP’s  policy  to  share  with  others  through  publications,  presentations,  and  discussions  its 
evaluation  plan,  methods,  tools,  data  (excepting  proprietary  data),  and  results. 

2.2  Internal  Demands  for  Evaluation  of  ATP 

There  also  are  strong  internal  demands  for  evaluation  of  the  ATP.  These  demands  come  from 
within  ATP  itself,  as  well  as  from  its  parent  departments:  the  National  Institute  of  Standards  and 
Technology  (NIST),  the  Technology  Administration,  and  the  U.S.  Department  of  Commerce. 
Internally,  the  ATP  is  interested  in  performance  evaluation,  first,  as  a management  tool  to  make 
the  program  better  meet  its  mission  and  operate  more  efficiently;  second,  to  find  out  how  we  are 
doing;  and,  third,  to  meet  the  many  external  requirements  and  requests  for  ATP  program  results. 
Finding  out  as  early  as  possible  what  is  working  best  and  what  is  producing  results  can  inform  the 
project-selection  phase  of  the  ATP,  improve  the  management  of  the  program,  and  likely  increase 
the  long-run  success  of  the  ATP  in  meeting  its  economic  and  technical  goals.  Determining  if  the 
ATP  is  in  the  long-run  meeting  its  goals  and  realizing  a large  return  for  the  U.S.  taxpayer  is,  of 
course,  essential  to  informing  technology  policy. 

2.3  Mandated  Requirements  for  Evaluation 

Title  II  of  the  American  Technology  Preeminence  Act  of  1991  (P.L.  102-245,  enacted  in  1992) 
directed  that  a comprehensive  report  on  the  results  of  the  ATP  be  submitted  to  each  House  of  the 
Congress  and  the  President  not  later  than  1996.  (This  report,  The  Advanced  Technology 
Program:  A Progress  Report  on  the  Impacts  of  an  Industry-Government  Technology 
Partnership , was  delivered  in  April  1996.) 

In  addition,  the  ATP,  like  other  federal  programs,  is  subject  to  the  evaluation  requirements  of  the 
1993  Government  Performance  and  Results  Act  (GPRA).  The  GPRA  resulted  from  a bipartisan 
effort  to  improve  accountability,  productivity,  and  effectiveness  of  Federal  programs  through 
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strategic  planning,  goal  setting,  and  performance  assessment.  The  ATP/NIST,  like  other  Federal 
government  agencies,  is  developing  assessment  plans  and  techniques,  and  carrying  out  evaluation 
studies  in  compliance  with  the  GPRA. 

2.4  What  and  When  to  Measure 

The  three  dimensions  of  ATP  represented  in  figure  1 — funding  projects  that  (1)  add  to  the 
scientific  and  technical  knowledge  base,  (2)  result  in  and  hasten  technology-based  commercialized 
products  and  processes,  and  (3)  ultimately  yield  widespread  economic  benefits  beyond  the  direct 
awardees  — condition  and  inform  ATP’s  evaluation  plan.  The  program’s  dimensions  inform  the 
decisions  of  what  to  evaluate,  when  the  evaluation  can  occur,  and  how  it  can  be  done.  Also 
important  to  designing  ATP’s  evaluation  plan,  but  not  depicted  in  the  static  model  of  figure  1,  are 
the  process  dynamics  within  each  dimension  and  at  the  points  of  transition  from  one  dimension  to 
another. 

The  fact  that  the  objective  of  the  ATP  is  to  maximize  “Q,”  rather  than  any  single  one  of  the  three 
program  dimensions  means  that  the  metrics  in  each  dimension  cannot  fairly  be  taken  in  isolation  of 
the  others.  For  example,  the  ATP  is  charged  not  with  selecting  projects  to  maximize 
contributions  to  the  scientific  and  technical  knowledge  base,  but  rather  projects  that  press  the 
state  of  the  art  while  satisfying  the  other  two  dimensions  of  the  program.  Thus,  one  target  of 
program  metrics  is  the  extent  to  which  the  research  funded  by  ATP  adds  to  the  nation’s  scientific 
and  technical  knowledge  base,  but  the  interpretation  of  results  must  be  made  within  the  context  of 
the  other  dimensions:  What  is  the  likelihood  that  award  recipients  will  pursue  commercialization 
of  the  resulting  technology  after  the  research  is  completed,  and  what  is  the  likelihood  that  there 
will  be  substantial  spillover  benefits  to  the  nation  not  appropriated  by  the  companies  carrying  out 
the  project?  If  one,  on  the  other  hand,  focuses  on  the  extent  to  which,  and  speed  with  which,  the 
awardees  translate  their  new  technical  know-how  into  products,  processes,  and  services  with 
commercial  significance,  it  is  important  to  take  into  account  the  degree  of  innovation  entailed  and 
appropriability  problems.  Similarly,  if  the  focus  is  spillover  benefits,  one  needs  also  to  consider 
private  benefits.  And  one  always  needs  to  question  what  the  level  of  benefits  would  likely  be 
without  the  ATP.  No  single  program  dimension  is  alone  the  end-all  of  ATP  or  the  focus  of  its 
program  metrics.  Measuring  the  level  of  “Q”  is  the  quest  of  program  metrics,  but  that  is,  of 
course,  a complex  and  exceedingly  difficult  challenge  that  we  are  not  likely  to  achieve  except 
piecemeal,  in  stages  and  by  degrees. 

Not  only  do  these  three  dimensions  serve  to  constrain  one  another,  but  they  are  also  linked 
dynamically  in  cause-and-effect,  tension-bound,  and  time-dimensioned  relationships.  Projects  that 
push  further  on  the  innovation  envelope,  for  example,  may  present  greater-than-average 
marketplace  challenges  to  awardees  in  achieving  their  commercialization  goals.  Failure  to  achieve 
commercialization  goals  in  a timely  way  may  jeopardize  the  achievement  of  most  of  the  targeted 
spillover  benefits.  The  cast  of  characters  needed  to  accomplish  technical  goals  is  likely  different 
from  those  needed  to  achieve  business  and  economic  goals.  Innovators  working  with  certain 
technologies  in  industries  with  certain  characteristics  are  more  likely  to  achieve  spillover  benefits 
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than  others.  Difficult-to-capture  intellectual  property  in  the  technology  may  weaken  the  chances 
of  commercial  follow-through,  but  strengthen  the  chances  of  spillover  benefits.  In  short,  there  is 
much  to  understand  about  project  paths,  transition  points,  diffusion  patterns  and  rates,  and 
underlying  cause-and-effect  relationships  that  combine  to  determine  program  outcomes. 

Ideally,  bottom-line  metrics  for  long-term  program  impact  will  include  measures  of  private  rates 
of  return  on  research  investment,  social  rates  of  return,  and  “public  rates  of  return,”  that  is,  the 
social-rate-of-retum  component  attributable  to  the  ATP.  But,  realistically,  it  is  expected  that 
proposed  evaluation  studies  will  tackle  parts  of  the  problem;  not  all  of  them  at  once.  Acceptable 
projects  may  address  only  one  of  the  three  dimensions,  or  only  certain  aspects  of  project  paths, 
transition  points,  diffusion  patterns,  and  the  like. 

2.5  ATP’s  Evaluation  Strategy 

To  square  the  often  pressing  demands  in  the  short  run  for  evaluation  results  with  the  reality  that 
patience  is  required  to  realize  and  validate  empirically  long-run  program  outcomes,  the  ATP  has 
adopted  a time-dimensioned,  multi-part  evaluation  strategy.  This  strategy  has  thus  far  entailed  (1) 
the  use  of  short-run  indicators  of  program  progress  towards  targeted  long-term  goals;  (2) 
descriptive  project  progress  updates;  (3)  case  studies  of  projects  and  groups  of  projects;  (4) 
projections  of  expected  long-term  impacts  conducted  ex  ante  of  actual  impacts  (with 
acknowledgment  of  the  limitation  imposed  by  the  large  uncertainties  entailed  in  such  projections); 
(5)  surveys  of  company  plans,  technical  and  business  progress,  and  assessments  of  short-term 
business  impacts;  and  (6)  statistical,  descriptive  profiling  of  projects,  participants,  technologies, 
and  intended  applications  of  the  technologies.  At  the  same  time,  the  ATP  is  working  to  improve 
the  tools  of  longer  term  evaluation,  to  collect  critical  data  that  will  be  needed  to  implement  this 
activity,  to  build  feedback  loops  between  project  selection  and  project  evaluation,  and  in  other 
ways  to  enlarge  and  improve  its  capabilities  both  of  achieving  the  long-run  technological  and 
macroeconomic  goals  of  the  program  and  of  measuring  specific  performance. 

2.6  Evaluation  Techniques  Currently  Used  by  the  ATP 

Peer  review  of  proposed  projects  against  project  selection  criteria  is  applied  up-front  in  ATP’s 
rigorous  awards  competitions.  The  goal  is  to  select  projects  that  are  likely  to  achieve  the  ATP’s 
mission.  It  is  sometimes  overlooked  that  this  step  is  a form  of  project  evaluation,  because  the 
assessment  occurs  gx  ante,  with  the  emphasis  on  achieving  the  program’s  mission  as  opposed  to 
measuring  performance  towards  achieving  the  mission.  At  this  initial  awards  stage  of  the  ATP 
project  cycle,  “expert”  evaluators  are  charged  with  using  their  resident  knowledge  and  judgment 
to  evaluate  proposals  against  published  selection  criteria  and  to  identify  those  that  appear  to  have 
strong  potential  for  achieving  a high  “Q”  value  as  illustrated  schematically  in  figure  1 . Scientists, 
engineers,  business  people,  and  economists  — all  of  whom  agree  to  abide  by  non-disclosure  and 
avoidance  of  conflict-of-interest  rules  and  sign  statements  to  that  effect  --  serve  as  peer  reviewers. 
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The  opportunity  exists  to  feed  back,  as  appropriate,  the  results  of  performance  evaluation  to  the 
peer-reviewers,  with  the  objective  of  informing  the  initial  project  selection  stage  so  that  future 
success  conditions  are  enhanced.  A step  in  this  direction  would  be  a more  explicit  evaluation  of 
the  spillover  potential  of  proposed  projects  by  comparing  project/proposer  attributes  against  an 
identified  set  of  factors  that  are  found  likely  to  increase  or  decrease  spillover  effects.8 

Real-time  project  monitoring  is  used  by  ATP  to  determine  the  progress  of  funded  projects 
currently  underway  against  technical,  business,  and  economic  goals.  This  is  also  sometimes 
overlooked  as  entailing  a component  of  project  evaluation,  perhaps  in  part  because  the  project 
monitoring  task  also  encompasses  project  management  and  administrative  activities,  and  in  part 
because  it  provides  just  pieces  of  the  information  needed  for  evaluation.  Project  monitoring, 
however,  is  important  to  evaluation,  in  that  it  brings  ATP  staff  face-to-face  with  the  performing 
entities  and  provides  the  opportunity  for  the  staff  to  become  intimately  familiar  with  the  projects, 
particularly  the  technical  work.  Thus  informed,  the  project  monitoring  staff9  often  serve  as 
informational  sources  for  project  evaluation  case  studies. 

Data  collection  and  analysis  is  used  by  the  ATP  to  track  project  progress,  to  understand  the 
overall  project  portfolio  statistically  , to  assess  results,  and  to  contribute  eventually  to  measuring 
long-run  outcomes.  Data  come  primarily  from  third-party  surveys,  ATP  special  studies,  and 
ATP’s  internally  administered  “business  reporting  system.” 

Case  studies  of  ATP-fiinded  projects  and  groups  of  related  projects  have  been  performed  at 
various  stages  of  the  project  life  cycle  to  capture  progress  to  date,  measure  short-to-medium  term 
impacts  at  the  level  of  the  firm,  and,  in  some  cases,  provide  the  information  needed  to  interface 
with  macroeconomic  models  for  projecting  national  impacts. 

Surveys  of  ATP  project  participants  have  been  used  to  capture  and  report  statistically  on  short- 
run  results  — particularly  early  business-related  progress,  as  well  as  to  gather  feedback  from 
program  participants  on  their  satisfaction  with  working  with  the  ATP. 

Econometrics  and  other  statistical  analysis  techniques  are  being  used  by  the  ATP  to  shed  light 
on  underlying  cause-and-effect  relationships,  such  as  spillover  mechanisms,  and  to  project  impacts 
from  the  firm  level  across  the  entire  economy  using  large-scale  macroeconomic  models. 

Modeling  has  been  used  to  provide  frameworks  for  better  understanding  and  assessing  the 
program. 


8For  example,  ATP’s  ability  to  assess  a proposed  project’s  likelihood  of  yielding  spillover 
effects  is  expected  to  be  assisted  by  a recent  report  by  Adam  Jaffe,  Economic  Analysis  of 
Research  Spillovers;  Implications  for  the  Advanced  Technology  Program,  Draft  Report,  October 


1996. 


9ATP  uses  the  descriptor,  “Project  Manager,”  for  this  staff  oversight  function. 
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2.7  Overview  of  ATP’s  Evaluation  Accomplishments  through  August  1996 

2.7.1  Evaluation  Planning 

The  ATP’s  Economic  Assessment  Office  has  developed  ATP’s  approach  to  economic  evaluation 
in  consultation  with  leading  economists  in  the  field.  The  Economic  Assessment  Office  in 
conjunction  with  the  National  Bureau  of  Economic  Research  has  held  periodic  workshops  with 
leaders  in  evaluating  technological  impacts  to  obtain  feedback  on  the  ATP’s  evaluation  approach 
and  plans,  and  to  solicit  advice  on  future  directions  (Workshops  chaired  by  Professor  Zvi 
Griliches  of  Harvard  University  were  held  at  NIST  in  December  1994  and  September  1995). 

Input  to  the  ATP  from  those  workshops,  as  well  as  from  other  sources,  is  reflected  in  the  outline 
of  topics  of  interest  given  in  section  3. 

An  early  background  planning  study  was  performed  for  the  ATP  by  Albert  Link,  consultant  (A. 
Link,  Measuring  the  Economic  Impact  of  the  Advanced  Technology  Program;  A Planning  Study, 
January  1992).  ATP’s  evaluation  plan  has  been  described  in  internal  documents  and  in  a number 
of  internal  and  external  presentations  (e.g.,  R.  Ruegg,  “R&D  Evaluation:  Methodological 
Issues ,”  American  Evaluation  Association  1994  Annual  Meeting),  and  summarized  in  three  NIST 
special  reports  (Setting  Priorities  and  Measuring  Results  at  the  National  Institute  of  Standards 
and  Technology,  January  31,  1994;  Delivering  Results,  June  1995;  and  The  Advanced 
Technology  Program:  A Progress  Report  on  the  Impacts  of  an  Industry-Government  Technology 
Partnership,  April  1996).  The  evaluation  plan  is  updated,  expanded,  and  presented  in  more  detail 
in  a forthcoming  ATP  report  (R.  Ruegg,  The  Advanced  Technology  Program  ’ s Economic 
Evaluation  Plan  and  Progress  in  Implementation,  draft  report  in  preparation).  The  3-D  model  of 
the  ATP  presented  and  discussed  earlier  has  provided  a useful  framework  for  recent  planning  of 
program  evaluation.  (J.-C.  Spender,  The  Three  Dimensional  Model  of  the  Advanced  Technology 
Program,  September  1996  draft  report). 

2.7.2  Data  Collection  and  Analysis 

The  Economic  Assessment  Office  has  established  several  databases  in  support  of  program 
evaluation.  The  “Awards  Database”  is  used  for  statistical  profiling  of  funded  projects, 
technologies,  and  participants.  It  provides  answers  to  numerous  questions  about  what,  where, 
and  whom  ATP  is  funding  and,  therefore,  is  useful  as  a management  tool  and  as  a resource  for 
project  evaluation.  It  is  supplemented  by  an  “Applicants  Database”  --  confidential  because  the 
ATP  does  not  release  the  names  of  applicants,  only  awardees  --  but  potentially  useful  for 
evaluation.  An  integrated  set  of  databases,  called  the  “Business  Reporting  System”  is  an 
important  component  of  ATP’s  data  compilation  in  support  of  project  evaluation.  These  data, 
compiled  systematically  by  electronic  survey  of  project  participants,  track  the  evolution  of 
projects  towards  achieving  their  business  and  economic  goals.  The  Business  Reporting  System 
consists  of  several  parts:  An  initial  report  on  planned  application  areas  for  the  technology  and 
planned  strategies  for  commercialization;  quarterly  short  reports  that  update  participant  status 
and  call  out  major  business  developments  during  the  period;  and  an  annual  report  on  progress 
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towards  implementing  the  commercialization  strategies  and  on  short-term  economic  impacts  of 
the  projects,  including,  but  not  limited  to,  early  sales  revenues,  impacts  on  R&D  cycles, 
collaboration  effects,  intellectual  property  creation,  and  company  changes  in  jobs  they  attribute  to 
their  ATP  project.  Additional  sections  of  the  Business  Reporting  System  now  under  development 
are  a close-out  report  which  completes  the  reporting  of  developments  during  the  course  of  the 
research  project,  and  updates  future  plans  for  commercialization;  and  a post-project  report,  to  be 
administered  three  times  (every  other  year)  over  the  six-year  period  following  project  completion, 
to  capture  future  commercialization  efforts  and  data  supporting  assessment  of  technology 
diffusion  and  spillover  effects.  (In  addition,  quarterly  technical  reports  are  filed  with  the  ATP 
for  each  project  during  its  ATP-funding  life,  but  these  are  unique  for  each  project  and  the 
information  does  not  lend  itself  to  database  format  and  statistical  analysis.) 

The  Business  Reporting  System  has  been  implemented  for  all  projects  funded  by  ATP  since  FY 
93.  At  present,  there  are  450  organizations  in  the  database,  representing  about  200  projects. 
(Approximately  sixty  projects  funded  prior  to  FY  93  are  not  in  the  Business  Reporting  System, 
but  partial  data  have  been  captured  for  some  of  these  projects.)  A recent  ATP  staff  paper 
analyzed  some  of  the  available  data  from  the  Business  Reporting  System  and  reported  early 
results  (J.  Powell,  The  ATP’s  Business  Reporting  System:  A Tool  for  Economic  Evaluation, 
September  1996).  Additional  databases  capture  data  resulting  from  non-routine,  in-house  studies 
by  ATP  staff,  and  intellectual  property  developed  in  the  ATP-funded  projects. 

2.7.3  Broad  Surveys  of  Project  Participants 

The  ATP  has  sponsored  two  broad  surveys  of  funded  companies  by  third-party  contractors  using 
telephone  interviews.  The  first  survey  included  26  organizations  participating  in  the  eleven 
projects  funded  by  ATP  in  its  first  competition,  one  year  into  the  projects  (Solomon  Associates, 
The  Advanced  Technology  Program;  An  Assessment  of  Short-Term  Impacts:  First  Competition 
Participants,  February  1993).  The  second,  larger  survey  included  125  companies  and  consortia 
who  participated  in  the  ATP  from  1990  to  1992  (Silber  and  Associates,  Survey  of  Advanced 
Technology  Program  1990-1992  Awardees:  Company  Opinion  about  the  ATP  and  Its  Early 
Effects,  January  30,  1996).  At  the  time  of  the  second  survey,  most  of  the  participants  covered 
were  well  into  their  research  projects;  a few  had  reached  completion. 

2.7.4  Background  Studies  on  Evaluation 

Edwin  Mansfield  prepared  a background  paper  for  the  ATP  on  estimating  social  and  private 
returns  from  innovations,  preparatory  to  performing  a set  of  project  case  studies  (E.  Mansfield, 
Estimating  Social  and  Private  Returns  from  Innovations  Based  on  the  Advanced  Technology 
Program:  Problems  and  Opportunities,  January  1996.  Professor  Adam  Jaffe  prepared  a 
background  report,  now  in  ATP  review,  on  spillover  effects  (A.  Jaffe,  Economic  Analysis  of 
Research  Spillovers:  Implications  for  the  Advanced  Technology  Program,  October  1996).  The 
report  models  spillover  effects  and  identifies  factors  that  influence  whether  a given  project  is  more 
or  less  likely  to  lead  to  large  spillovers.  ATP’s  staff  has  also  published  general  methodological 
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pieces  on  economic  evaluation  (R.  Ruegg,  “Economic  Evaluation  Methods,”  Handbook  on 
Energy  Economics,  1996;  and  “Risk  Assessment,”  The  Engineering  Handbook,  1996). 

2.7.5  Case  Studies 

Several  case  studies  of  ATP  projects,  done  in  the  early  years  of  ATP  by  Professor  Albert  Link, 
focused  on  research  efficiency  in  research  joint  ventures,  and  several  additional  studies  that  update 
those  earlier  cases  are  nearing  completion  (A.  Link,  Economic  Analysis  of  the  Printed  Wiring 
Board  Research  Joint  Venture,  September  1996  draft  report,  updates  his  earlier  case  study  of  the 
same  joint  venture  project  published  in  April  1993). 

The  ATP  is  experimenting  with  combining  microeconomic  case  study  with  the  use  of 
macroeconomic  models  to  develop  national  economic  impact  projections  for  projects.  Two 
studies  taking  two  different  approaches  to  using  the  REMI  model  for  macroeconomic  projections 
are  nearing  completion:  (CONSAD  Research  Corporation,  The  Application  of  a Macroeconomic 
Interindustry  Model  toon  ATP  Joint  Venture  Project;  A Case  Study  of  the  Development  of 
Advanced  Technologies  and  Systems  for  Controlling  Dimensional  Variation  in  Automobile  Body 
Manufacturing  (the  2 millimeter  Project),  July  1996  draft  report;  and  E.  Robles,  Using  the  REMI 
Model  to  Estimate  the  National  Economic  Impacts  of  the  2 millimeter  Project,  October  1996 
draft  report). 

2.7.6  Studies  of  ATP  Counterpart  Programs 

The  ATP  collects  information  on  foreign  programs  that  are  similar  to  ATP  with  two  objectives: 

(1)  it  hopes  to  learn  from  the  experience  of  those  programs,  and  (2)  it  needs  data  from  them  to 
perform  “determinations  of  eligibility”  of  foreign-owned  U.S.  subsidiaries  to  participate  in  the 
ATP.  One  factor  in  the  eligibility  is  whether  U.S.  companies  are  treated  the  same  as  any  other 
company  in  ATP-similar  programs  that  operate  in  the  country  of  origin  of  the  ultimate  parent 
company  of  the  U.S.  subsidiary  that  is  applying  to  the  ATP.  An  ATP  report  in  progress  compares 
the  features  of  a number  of  counterpart  programs  with  the  ATP  (C.  Chang,  A Comparison  of  the 
U.S.  Advanced  Technology  Program  with  Similar  Programs  Abroad,  October  1996  draft  report). 


2.7.7  Other  Impact  Studies 

A study  based  on  participant  interview  data  was  recently  conducted  of  ATP’s  impact  on 
accelerating  technology  development  by  cutting  the  time  for  starting  projects  and  by  compressing 
research  cycle  time.  The  report  is  in  publication  (F.  Laidlaw,  “Acceleration  of  Technology 
Development  by  The  Advanced  Technology  Program:  The  Experience  of  28  Projects  Funded  in 
1991,  National  Institute  of  Standards  and  Technology,”  October  1996  draft  report).  Cycle-time 
reduction  is  of  keen  interest  to  the  ATP  because  ATP’s  authorizing  legislation  calls  for  it  to 
accelerate  R&D  and  the  commercialization  of  resulting  technologies. 
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3.  Evaluation  Research  Topics  of  Interest  to  the  ATP 


The  ATP  wishes  to  improve  and  extend  its  evaluation  capabilities,  tools,  and  metrics.  The  areas 
listed  below  are  of  special  interest.  Researchers  who  wish  to  propose  studies  in  these  areas  are 
reminded  to  relate  the  proposed  research  directly  to  the  ATP  and  to  consider  that  the  ATP  is 
interested  in  specific  implications  of  the  research  for  the  program.  Proposed  research  is  expected 
to  assist  in  the  measure  of  ATP’s  impacts,  and  may  also  inform  ATP  funding  decisions.  The  ATP 
considers  it  outside  its  scope  to  fund  general  studies  in  these  areas,  not  of  direct  relevance  to  it. 

3.1  Spillover  Pathways 

- The  role  of  market  forces  and  market  structures  in  determining  the  magnitude  of 
spillovers  from  ATP  projects,  including  market  and  knowledge  spillovers 

- Attributes  of  technologies  that  affect  the  magnitude  of  market  and  knowledge  spillovers 

- Empirical  relationships  between  social  and  private  rates  of  return 

- Studies  of  alternative  appropriation  mechanisms  and  their  implications  for  spillover 
benefits  in  different  industries  and  technological  areas,  and  as  employed  by  different 
organizational  types 

- Intra-industry  and  inter-industry  diffusion  mechanisms,  patterns,  and  rates,  and  their 
relationship  to  industry  structure,  technology,  and  other  factors 

- Bibliometric  studies  of  papers  and  patents  resulting  from  ATP-funded  projects 

3.2  Research  Collaborations 

- The  role  of  ATP  in  fostering  collaborations  and  the  benefits  of  collaborations 

- Joint  ventures  as  a means  of  internalizing  among  firms  the  spillovers  from  technology 
development 

- Joint-venture  structure  as  it  bears  on  the  willingness  and  capability  of  companies  to 
undertake  high-risk,  enabling  research  projects  and  carry  them  to  successful  outcomes 

3.3  Financing  Issues 

- Private  technology  investment  decisions  in  the  face  of  various  levels  of  uncertainty  and 
risk,  capital  requirements,  appropriation  mechanisms,  risk-adjusted  private  rate-of-retum 
hurdle  rates,  industry  sector,  firm  size,  technology  focus,  and  other  relevant  factors 
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- The  effect  of  ATP  project  funding  on  the  amount,  rate,  type,  and  scope  of  private 
investment  in  R&D 

- Effects  of  cost-sharing  rules 

- Industry  responses  to  direct  funding  for  R&D  (such  as  is  provided  by  the  ATP)  versus 
indirect  funding  (such  as  is  provided  by  tax  incentives) 

3.4  Economic  Modeling  and  Methodology 

- Development  of  new  and  improved  qualitative  and  quantitative  models  for  measuring 
economic  impacts  of  publicly  funded,  privately  executed,  enabling  technological  advances 

- Improvements  in  the  ability  to  translate  project-level  innovations  into  national  economic 
effects  by  the  use  of  large-scale  macroeconomic  models 

- Improved  modeling  and  understanding  of  the  innovation  gap  that  ATP  was  designed  to 
fill:  modeling  of  public/private  partnership  approaches  to  supplying  technologies  that  are 
neither  purely  private  goods  nor  purely  public  goods  --  those  high  in  spillover  benefits  but 
with  insufficient  private  return  to  attract  total  private  funding,  i.e.,  the  “preeompetitive, 
generic”  technologies  that  lie  between  the  basic  research  of  the  university  lab  and  the 
typical  applied  work  of  a commercial  lab 

- Development  of  qualitative  and  quantitative  models  of  R&D  investment  decisions, 
including  models  of  the  impact  of  ATP  funding  on  private  sector  R&D  investment. 

3.5  Project  and  Program  Impact  Assessment 

- Extending  impact  assessment  to  include  the  long-run  economic  effects  of  organizational 
and  “cultural”  changes  that  may  result  from  the  ATP 

- Assessment  of  ATP’s  impact  on  acceleration  of  technology  development  and 
commercialization  in  different  technologies  and  industry  sectors 

- Measurement  of  direct  and  indirect  impacts  of  ATP-funded  technology  development  on 
productivity,  output,  jobs,  and  earnings 

- Measurement  of  spillover  impacts  of  ATP-funded  projects,  including  impacts  on 
customers,  competitors,  and  other  firms  inside  and  outside  the  innovating  industry  sector 
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- Case  studies  of  groups  of  interrelated  ATP  projects,  including  assessment  of  possible 
synergistic  effects  among  them,  and  the  estimation  of  private,  social,  and  public  rates  of 
return 

- Project  failure  analysis 

3.6  Other  Topics 

The  ATP  does  not  consider  the  above  listing  of  topics  necessarily  an  exhaustive  treatment.  The 
listing  is  intended  merely  to  convey  to  the  evaluation  community  a general  idea  of  the  current 
evaluation  interests  of  the  ATP.  It  is  expected  that  experts  working  in  the  area  will  likely  suggest 
research  topics  not  explicitly  referenced  here  that  nevertheless  might  be  of  great  value  to 
evaluating  the  ATP. 
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4.  ATP’s  Plans  for  Reviewing  and  Funding  Evaluation  Proposals 

4.1  Evaluation  of  Proposals  by  an  ATP-Convened  Board 

Approximately  at  six-month  intervals,  the  ATP  plans  to  convene  a Source  Evaluation  Board 
(SEB)  to  consider  proposals  submitted  to  the  ATP  by  outside  researchers  on  topics  pertaining  to 
the  evaluation  of  the  ATP’s  impacts  on  the  economy.  The  SEB,  comprised  of  government 
economists  and  technologists  in  and  outside  of  the  ATP,  will  review  each  candidate  proposal. 

The  SEB  will  evaluate  the  merits  of  the  proposed  study’s  objective  and  approach;  the  relevancy  of 
the  study  to  ATP’s  evaluation  concerns  as  outlined  in  section  3;  cost;  qualifications  of  the 
performer;  and  potential  obstacles  to  success.  With  the  advice  of  the  SEB,  the  ATP  will  decide 
whether  or  not  it  wishes  to  pursue  the  work  as  proposed,  and  what  modifications  to  suggest.  If  it 
wishes  to  support  the  work,  it  will  decide  the  most  appropriate  approach. 

4.2  Role  of  the  National  Bureau  of  Economic  Research  (NBER) 

Some  studies  proposed  to  the  ATP  and  evaluated  by  the  SEB  will  be  funded  as  task  orders  under 
an  “umbrella”  contract  with  the  NBER,  which  is  serving  as  a general  contractor  to  the  ATP  for 
the  purpose  of  undertaking  studies  that  press  the  state-of-the-art  of  economic  evaluation.  The 
NBER  will  be  asked  to  review  each  draft  task  order  submitted  to  it  by  the  ATP,  assess  the 
feasibility  of  undertaking  the  study  as  described,  decide  which  it  will  support,  and  recommend  any 
necessary  changes  to  those  it  agrees  to  undertake.  For  projects  funded  by  the  ATP  through  task 
orders  with  the  NBER,  the  NBER  will  serve  as  administrator,  tracking  progress  against  project 
milestones  and  monitoring  costs  in  conformance  with  the  terms  of  its  contractual  Statement  of 
Work  with  NIST.  Sign-off  approval  of  project  work  will  be  performed  jointly  by  the  President  of 
the  NBER,  or  the  NBER’s  ATP-Project  Director,  and  by  the  ATP. 

As  background,  the  NBER  is  a private,  nonprofit,  nonpartisan  research  organization  committed  to 
advancing  and  disseminating  unbiased  economic  research  among  public  policy  makers,  business 
professionals,  and  the  academic  community.  The  NBER  coordinates  research  by  more  than  380 
economics  professionals  at  leading  universities  and  business  schools  throughout  the  U.S., 
concentrated  in  four  areas  of  research:  developing  new  statistical  measurements,  estimating 
quantitative  models  of  economic  behavior,  assessing  the  effects  of  public  policies  on  the  U.S. 
economy,  and  projecting  the  effects  of  alternative  policy  proposals.  World  renowned  researchers 
in  the  field  of  the  economics  of  science  and  technology,  including  experts  in  evaluation,  are 
associate  members  of  the  NBER.  The  NBER  is  unique  in  that,  unlike  other  economic  research 
centers  whose  activities  consist  primarily  of  the  research  of  a relatively  small  resident  staff,  the 
NBER  has  links  to  hundreds  of  leading  professional  economists  with  a wide  range  of  interest  and 
expertise,  presenting  a great  range  of  resources  with  which  to  meet  the  ATP’s  evaluation  needs. 
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5.  Advice  on  Preparing  Evaluation  Proposals  and  Reports  for  the  ATP 

5.1  Desired  Proposal  Characteristics 

It  is  expected  that  successful  proposals  of  economic  research  to  the  ATP  will  exhibit  the  following 

characteristics: 

• The  proposed  research  will  be  directed  by  researchers  with  previously  demonstrated 
capability  in  undertaking  economic  research  of  the  type  proposed,  as  evidenced  by 
publications  and  citations  in  economics  periodicals  and  professionally  refereed  journals. 
Qualified  graduate  students  and  other  qualified  researchers  may  be  involved  in  a study. 

• The  research  will  pertain  directly  to  evaluation  issues  of  concern  to  the  ATP. 

• The  researcher  will  bridge  the  gap  between  a purely  theoretical  or  general  treatment  of  a 
topic  and  the  practical  implications  of  the  research  for  the  ATP. 

5.2  Desired  Proposal  Contents 

Proposers  of  economic  research  are  asked  at  a minimum  to  include  the  following  elements: 

• Executive  Summary  explaining  in  lay  terms  the  study  objective  and  why  it  is  important  (a) 
for  the  ATP  and  (b)  for  the  general  field  of  economic  evaluation  of  technology.  Explain 
what  will  likely  be  learned  and  how  the  ATP  is  likely  to  benefit  from  it.  Indicate  what  new 
methods,  tools,  or  data  are  likely  to  result,  and  their  significance  to  the  field  and  to  the 
ATP. 

• Background  synopsis  of  the  relevant  literature  and  description  of  current  state  of  the  art  or 
situation. 

• A description  of  the  proposed  evaluation  research,  indicating  the  methodology,  scope, 
data  requirements,  analytical  techniques  to  be  employed,  specific  research  tasks, 
hypotheses  to  be  tested,  and  relevancy  to  the  ATP. 

• Anticipated  opportunities  for  future  research  based  on  the  work  proposed. 

• Schedule,  outputs,  and  tentative  costs,  showing  a breakdown  by  categories. 

• Resumes  of  principal  researchers,  including  publications,  education,  and  work  experience. 

• Attachments  of  particularly  relevant  supporting  materials  at  the  proposer’s  discretion. 
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5.3  Criteria  Against  which  Evaluation  Proposals  will  be  Judged 

Proposals  will  be  judged  by  the  SEB  against  the  following  criteria: 

• Merits  of  the  proposed  objectives  and  soundness  of  approach 

• Relevancy  to  ATP’s  evaluation  concerns 

• Qualifications  of  the  researchers 

• Cost  and  timeliness 

5.4  Minimum  Requirements  for  Contractors 

Contractors  of  economic  research  for  the  ATP  will  at  a minimum  be  subject  to  the  following 

requirements: 

• Adherence  to  terms  regarding  time,  cost,  and  milestones  established  in  the  contract/task 
order. 

• Delivery  of  draft  reports  according  to  schedule. 

• Delivery  of  final  report  according  to  schedule,  subject  to  approval  by  the  ATP,  and,  if 
task-ordered  through  the  NBER,  by  the  President  of  the  NBER  or  NBER’s  ATP-project 
Director. 

5.5  Requirements  for  Evaluation  Reports 

Typically,  researchers  should  plan  on  submitting  several  drafts  of  their  reports  in  sequence,  with  a 

feedback  loop  for  comments  and  revision,  prior  to  the  final  report  submittal. 

Reports  would  generally  be  expected  to  include  the  following  elements: 

• Abstract. 

• Keywords  list. 

• Executive  Summary  explaining  the  study  objective  and  its  importance,  providing  a brief 
overview  of  the  research  approach,  a summary  of  the  principal  findings  listed  preferably  in 
bullet  style,  and  a brief  explanation  of  any  important  limitations  or  caveats  of  which  the 
reader  should  be  aware. 

• Problem  statement  and  background  information. 
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• Methodology  section. 

• Discussion  of  data  and  assumptions  if  used. 

• For  some  studies,  a review  of  the  literature. 

• Detailed  results  section. 

• Summary  and  Conclusions. 

• References/bibliography. 

• Tables  and  exhibits  if  used. 

Researchers  are  usually  asked  to  submit  both  a hard  copy  of  their  report  and  an  electronic  file 
copy.  Figures  and  tables  should  either  be  integrated  electronically  in  the  text  or  camera-ready 
hard  copy  should  be  provided. 

5.6  ATP  Contact  Information 

Questions,  comments,  or  proposals  on  evaluation  can  be  submitted  to  the  ATP  at  any  time  at  the 
following  address: 

Economic  Assessment  Office,  Advanced  Technology  Program 
National  Institute  of  Standards  and  Technology 
Administration  Building,  Room  A3  00 
Gaithersburg,  MD,  20899 

You  may  wish  to  contact  Rosalie  Ruegg,  Office  Director:  301-975-6135,  e-mail 
rosalie.ruegg@nist.gov;  Jeanne  Powell,  Group  Leader:  301-975-4196,  e-mail 
jeanne.powell@nist.gov;  Richard  Spivack,  Economist:  301-975-5063,  e-mail 
richard.spivack@nist.gov;  or  Connie  Chang,  Economist:  301-975-4318,  e-mail 
connie.chang@nist.gov.  The  Office  FAX  is  301-921-6319. 

In  support  of  its  technology  development  activities,  the  ATP  maintains  a toll-free  “hotline” 
number  1-800-ATP-FUND.  Application  kits  for  applying  to  ATP’s  announced  competitions  for 
awards  to  develop  advanced  technologies  (not  evaluation  research!)  may  be  requested  by  calling 
the  hotline  number.  The  ATP  also  makes  information  available  through  its  World  Wide  Web 
home  page  at  http://www.atp.nist.gov,  where  you  can  find  a cross  link  to  the  Economic 
Assessment  Office’s  home  page,  and  copies  of  some  of  the  evaluation  reports. 

Your  comments  are  welcomed. 
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